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(57) Abstract 

Monitoring device (100) including a processor (104), a television remote control unit signal detector (102), connected to the processor, 
a signal capturing unit (1 10), connected to the processor and a conununication interface (108) connected to the processor, wherein the 
television remote control unit signal detector (102) provides detected television remote control unit (134) signal to the processor, the signal 
capturing unit provides detected television signal to the processor, and wherein the processor transmits data related to the detected television 
signal and the detected television remote control unit signal to a remote monitoring center (120) via the communication interface. 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the firont pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spun 


LS 


Lesodio 


SI 


Slovenia 


AM 


Annenia 


FI 


Finland 


LT 


Littiuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Bnifcina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


UZ 


Uzbekistan 


CF 


Central African Republic 


JP 


Jtpan 


NE 


Niger 


VN 


Vici Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


ZW 


Zimbabwe 


CI 


Cdce d'l voire 


KP 


Demoomic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


FT 


Portugal 






CU 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Gennany 


U 


Lieditenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







wo 00/70869 



PCT/ILOQ/00278 

MONITORING SYSTEM 



FIELD OF THE INVENTION 

The present invention relates to a method and apparatus for 
monitoring video streams in general, and to a method and apparatus for 
monitoring television viewing habits, in particular. 

BACKGROUND OF THE INVENTION 

Television monitoring methods and systems are known in the 
art. Such systems are used to monitor the habits of television viewers and 
rate television channels and programs according to their popularity. 

Some of these systems employ remote control detectors which 
receive transmissions produced by a remote control unit associated with 
the viewed television set. 

Some of the methods known in the art employ special 
identification codes, which are embedded in the broadcast signal at a 
broadcasting center. The broadcasted signal is detected by a monitor, 
which is connected to the television set of a selected viewer. This monitor 
extracts the embedded identification code, thereby detemiining the 
currently viewed channel, and provides an indication to a monitoring 
center. The monitoring center accumulates infomnation from a plurality of 
such monitors and stores them in a database. The content of this 
database can then be used to statistically analyze viewing habits of a 
population. 

US Patent No. 5.235,414 to Cohen is directed to a system for 
unobtrusively monitoring the viewing habits of an audience to home 
entertainment systems, equipped with digital remote controls. This 
system ensures that any remote contrpi transmission is passed there 
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through before being provided to the destination device, which may be a 
television set, a stereo system, and the Wke. The monitor of the system 
conveys the detected remote control signal to various components of the 
home entertainment system, and is thus equipped to acquire and store 
audio/video tuning and channel selection data, for later transmission to a 

central computer. 

US patent No. 4,972,503 to Zurlinden, is directed to a system for 
determining television audience viewing habits, wherein the viewer tuning 
information is acquired from infrared, ultrasonic, or other control signals 
transmitted by a user through a television remote control. This system 
employs a channel detection apparatus which receives and processes the 
emitted tuning data, to determine the selected television channel. During 
this processing, the detection apparatus temporarily jams the remotely 
controlled receiver from responding to the emitted control signal. Once 
the given processing is substantially completed, the detection apparatus 
ceases the jamming, to permit the remotely controlled receiver to respond 
accordingly. 

The channel detection apparatus periodically rebroadcasts a 
control signal to the controlled receiver, instructing that receiver to tune 
the television to the last recorded channel. 

US patent No. 4,876,736 to Kiewit, is directed to a system for 
determining the channel to which a television receiver is tuned, by 
evaluating the signal emitted by that receiver's remote controller. The 
system employs a first remote control transmitter which is specially 
designed to facilitate the process of channel selection determination, that 
is effected before the selection-making control signal is passed on to the 
receiver. The first remote control transmitter is initialized using the 
existing remote control transmitter for the receiver. However, the existing 
remote control transmitter is not thereafter used. 
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US patent No. 5,532,732 to Yuen et al., is directed to a system 
for television audience monitoring. According to Yuen, an audience 
monitor is coupled to a television from which It receives the tuned 
television channel signal via an Input line. The audience monitor is 

5 actuated responsive to a television on/off signal, which It separately 
receives by one of two means -- through a line that passes the television's 
manual on/off switch, or through an infrared detector that receives the 
on/off command signal emitted by a remote controller. Upon command, 
the system detects a compressed identification which is embedded within 

10 the currently viewed channel. Then, the system transmits this 
identification to a remote location over a telephone line. 

US patent No. 5,382,970 to Kiefl, is directed to a system for 
monitoring and recording the viewing habits of television viewers. The 
system employs a plurality of personal data meters, each of which is 

15 equipped with, among other things, a microprocessor, an infrared 
detector, a cellular phone and modem (also referenced as a telephone 
module). Each personal data meter is earned by a viewer, such that when 
the viewer views a television program, the personal data meter receives 
the program selection information transmitted via the given television's 

20 remote control, and then derives the corresponding channel and time 
information. At a pre-selected time, the personal data meter transmits 
stored information to a central location through its cellular telephone 
module. The personal data meter also includes a confimiation button by 
which the user confinns the channel number that the data meter is set to. 

25 However, this is a feature merely for confirming the user's actual 
attendance. 

US patent No. 5,481,294 to Thomas et al., is directed to an 
audience measurement system for monitoring the viewing audience of 
television programs. The system includes a plurality of metering 
30 apparatuses distributed among statistically selected households. Each 
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metering apparatus includes tuning measurement equipment, having one 
or more sensors that acquire video and/or audio signals from a given 
television. The acquired signal(s) Is processed in such a manner that 
program information is derived from any ancillary code, which may be 
5 present in the acquired signal. If no ancillary codes are present, the 
characteristic signatures are extracted from the acquired signals, and 
compared to reference signatures, so as to passively determine the given 
program information. The resulting infonnation is transmitted wVn other 
pertinent data, through a telecommunications link to a central site. Each 
10 household metering apparatus, also includes a people meter which 
detects a signal transmitted by the given television's remote control. The 
stated purpose of the people meter, however, is merely to confirm the 
presence of audience members. 

US patent No. 4,697,209 to Kiewit, is directed to a system for 
15 identifying the programs being received by a television. Essentially, the 
system operates by extracting a signature from the given video signal 
upon detection of predetennined characteristics in that signal. The 
extracted signature and the times con-esponding thereto, are stored and 
subsequently compared, in order to identify the received television 
20 program. The detection of which channel/program is being viewed, is 
accomplished by using an embedded signature within the broadcast 
signal. 
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SUMMARY OF THE PRESENT INVENTION 

It is an object of the present invention to provide a novel method 
for monitoring television-viewing habits, which overcomes the 
disadvantages of the prior art. 

It is another object of the present invention to provide a novel 
device for monitoring television-viewing habits, which overcomes the 
disadvantages of the prior art. 

In accordance with the present Invention, there is thus provided 
a monitoring device, which includes a processor, a television remote 
control unit signal detector, connected to the processor, a signal-capturing 
unit, connected to the processor, and a communication Interface, 
connected to the processor. The television remote control unit signal 
detector, provides detected television remote control unit signal, to the 
processor. The signal-capturing unit provides the detected television 
signal to the processor. The processor transmits data related to the 
detected television signal, and the detected television remote control unit 
signal to a remote-monitoring center, via the communication interface. 

The television remote control unit signal detector can be an 
infra-red detector, a radio frequency detector, an ultrasound detector and 
the like. The signal capturing unit can be a microphone, a video camera, 
an audio interface, a video interface, a broadcast signal interface and the 
like. 

The data can Include objects such as a representation of the 
detected television signal, a representation of the detected television 
remote control unit signal, a time stamp, a digital signature determined 
from the detected television signal, channel identity information 
determined from the detected television remote control unit signal, and the 
like. 
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The device can further include a storage unit, connected to tlie 
processor, wherein the processor stores data in the storage unit for 
transmitting via the communication interface at a later time. 

According to one aspect of the invention, the signal capturing 
unit can include a tuner. Accordingly, the processor determines a 
broadcast channel from the detected television remote control unit signal, 
and provides a command to the tuner to select the broadcast channel. 
The tuner captures a broadcast signal transmitted on the broadcast 
channel. Finally, the processor determines a digital signature from the 
broadcast signal, and transmits the digital signature via the 
communication interface. 

According to another aspect of the invention, the device further 
includes a tuner, connected to the processor, and a comparator 
connected to the processor, the television interface and to the tuner. 
Accordingly, the processor determines a broadcast channel from the 
detected television remote control unit signal, and provides a command to 
the tuner to select the broadcast channel. The tuner captures a broadcast 
signal transmitted on the broadcast channel, and provides the broadcast 
signal to the comparator. The signal capturing unit provides the detected 
television signal to the comparator. Finally, the comparator compares 
respective properties of the detected television signal and the broadcast 
signal, and provides a match result to the processor for verifying that the 
determined broadcast channel corresponds to the detected television 
signal. 

According to a further aspect of the invention, the signal 
capturing unit can detect a television video signal, and provide the 
detected television video signal to the processor. Hence, the processor 
detects a plurality of frames in the detected television video signal. 

The device can further include a multi channel tuner, connected 
to the processor. In this case, the processor determines a plurality of 
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broadcast channels from the detected television remote control unit signal, 
and provides a command to the tuner to select the broadcast channels. 
The tuner captures a plurality of broadcast signals transmitted on the 
selected broadcast channels, and provides the broadcast signals to the 
processor. Finally, the processor compares between an image provided in 
each of the frames, and a selected one of the broadcast signals. 

In accordance with another aspect of the present invention, 
there is thus provided a method for detecting user selection of a television 
channel. The method includes the steps of detecting a remote control 
device transmission, receiving a television signal from a television set, and 
processing the television signal, thereby generating a signature therefrom. 

The method can further include the step of producing a record, 
which includes a plurality of data objects, wherein at least one of the data 
objects includes the signature, and at least another of the data objects 
includes a time stamp. The method can further include the step of 
transmitting the record to a remote control center. 

The method can further include the step of comparing the 
signature with at least one of a plurality of reference signatures, thereby 
determining a compatible reference signature therefrom. 

The method can further include the step of determining a stream 
corresponding with the compatible reference signature. The method can 
further include the step of transmitting information relating to the 
detennined stream, to a remote monitoring center. 

The method can further include the step of storing information 
relating to the determined stream. The method can further include the step 
of storing the signature. 

The method can further include the steps of detecting a video 
signal from a specific channel, processing the video signal, thereby 
generating a channel signature therefrom, comparing the channel 
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signature with the signature, thereby verifying that the specific channel 
corresponds to the television signal. 

The method can further include the step of selecting the specific 
channel according to the remote control device transmission. The method 
5 can further include the step of selecting a different channel, when the 
specific channel Is not compatible with the television signal. 

The method can further include the step of transmitting 
parameters relating to the specific channel, when the specific channel is 
compatible with the television signal. 
10 The method can further Include the steps of analyzing the 

television signal, thereby determining a plurality of frames, and retrieving a 
frame television signal from each of the frames. 

In accordance with a further aspect of the invention, there Is 
thus provided a method for detecting user selection of television channels. 
15 The method Includes the steps of receiving a television signal from a 
television set, detecting a plurality of channel Images within the television 
signal, and processing the television signal thereby generating a signature 
for each of the channel Images. 

The method can further Include the steps of detecting a remote 
20 control device transmission, and analyzing the remote control device 
transmission, so as to extract multiple channel commands embedded 
therein, wherein the multiple channel commands are further used in the 
step of detecting the plurality of channel Images. 

The method can further include the step of producing a plurality 
25 of records. Each of these records includes a plurality of data objects. At 
least one of the data objects Includes the signature, and at least another 
of the data objects includes a time stamp. The method can further include 
the step of transmitting the records to a remote control center. 

The method can further include the step of comparing each of 
30 the signatures with at least one of a plurality of reference signatures, 
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thereby determining at least one compatible reference signature, 
therefrom. 

The method can further include the step of determining a stream 
corresponding with the compatible reference signatures. 

The method can further include the steps of transmitting 
information relating to the determined stream, to a remote monitoring 
center. The method can further include the step storing information 
relating to the determined stream. The method can further include the step 
of storing the signature. 

The method can further include the steps of detecting video 
signals from a plurality of specific channels, processing the video signals 
thereby generating channel signatures therefrom, and comparing the 
channel signatures with the signature, thereby verifying that at least one of 
the specific channels corresponds to at least a portion of the television 
signal. 

The method can further Include the step of selecting the specific 
channels according to the remote control device transmission. 

The method can further include the step of selecting a different 
channel when at least one of the specific channels is not compatible with 
at least a portion of the television signal. 

The method can further include the step of transmitting 
parameters relating to the specific channel, when the specific channel is 
compatible with at least a portion of the television signal. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be understood and appreciated more 
fully from the following detailed description taken in conjunction with the 

drawings in which: 

Fig. 1 Is a schematic illustration of a television set, a broadcast 
center, a monitoring center and a device for monitoring a video signal, 
constructed operative in accordance with a preferred embodiment of the 

present invention; 

Figure 2 is a schematic illustration of a method for operating the 
combination of device and monitoring center of Figure 1 ; 

Figure 3 is a schematic illustration of a television set, a 
broadcast center, a monitoring center, and a television monitoring device, 
constructed and operative in accordance with another preferred 
embodiment of the present Invention; 

Figure 4 Is a schematic illustration of a method for operating 
monitoring device and monitoring center of Figure 3, operative in 
accordance with a further prefen-ed embodiment of the present Invention; 

Figure 5 is a schematic Illustration of a device, constructed and 
operative in accordance with another prefen-ed embodiment of the present 
invention; 

Figure 6 is a schematic illustration of a method for operating the 
device of Figure 5, operative in accordance with a further preferred 
embodiment of the present invention; 

Figures 7A and 7B are schematic illustrations of a television, 
which is adapted to provide multiple channel presentation; 

Figure 8 is a schematic illustration of a device for monitoring 
multiple channels, constructed and operative in accordance with another 
prefen-ed embodiment of the present invention; 
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Figure 9 is a schematic illustration of a method for operating the 
monitoring device of Figure 8, operative in accordance with a further 
preferred embodiment of the present invention; 

Figure 10 is an Illustration of a television set and two capturing 
devices, constructed and operative in accordance with another prefen-ed 
embodiment of the present invention; 

Figure 11A is a schematic illustration of the view detected by 
one of the cameras of Figure 10; and 

Figure 11B Is a schematic illustration of the view detected by 
another one of the cameras of Figure 10. 
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DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

The present invention overcomes the disadvantages of the prior 
art by combining stream monitoring with user action detection. In general, 
the present invention includes two main elements, one being a signature 
generation procedure and the other, a user operation detection procedure. 

The signature generation procedure processes the raw video 
signal and generates a signature therefrom. This signature is then 
compared with the signature produced from a reference stream to 
determine a match there between. Such a signature generation procedure 
is described in detail in US patent No. 5,790,236 to Hershtik et al., entitled 
"Movie processing system'*, which is hereby incorporated by reference. In 
this reference, the term signature is replaced with the term 
frame-characterizing information. 

The user operation detection procedure employs a plurality of 
novel procedures and devices to determine user actions. The two 
elements are combined throughout the present invention, in various 
degrees of integration and relationship. For example, the user operation 
detection procedure can be used to determine which reference stream 
should be used, as will be described in detail herein below. In the detailed 
description herein below, the term "remote control unit" can refer to 
"remote control device", and vice versa. 

Reference is now made to Fig. 1, which is a schematic 
illustration of a television set, a broadcast center, a monitoring center and 
a device for monitoring a video signal, generally referenced 100, operative 
in accordance with a preferred embodiment of the present invention. 

Device 100 includes a processor 104, a storage unit 106, a 
detector 102, a communication interface 108 and a microphone 110. The 
processor 104 is connected to storage unit 106, detector 102, 
communication interface 108 and to microphone 110. 
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Television set 130 receives commands from a user using a 
remote control unit 134. These commands can include change channel 
commands, open and shut down commands, attribute (volume, contrast 
brightness etc.) commands and the like. It is noted that the remote control 
unit 134 can transmit the user commands to the television set, via either 
wireless link such as Infra-red (IR), radio frequency (RF) or ultra-sound 
(US). Detector 102 is adapted according to the type of wireless media 
used by the remote control. In the present example, remote control unit 
134 utilizes IR transmission (generally referenced 136). Hence, detector 
102 of device 100. an IR detector. 

Television set 130 is connected to a broadcast center 124 via a 
video link 126. It is noted that video link 126 can include a cable TV link, 
an antenna, a satellite link, and the like. The broadcast center 124 
transmits a plurality of channels to the television 130. The user operates 
the remote control unit 134 so as to select which of the channels he 
wishes to see on the television 130. The television 130 includes a detector 
132 for detecting the signal 136 of the remote control unit 134. The signal 
136 includes a sequence of commands, according to the operations of the 
user. It is noted that the microphone 110 can be carried by the user, while 
wirelessly transmitting detected audio signals to the device. 

Reference is further made to Figure 2, which Is a schematic 
illustration of a method for operating the combination of device 100 and a 
monitoring center 120. 

As stated above, the device 100 detects the signal 136 of the 
remote control unit 134, using the detector 102 (step 150). The detector 
102 provides the detected signal to the processor 104, which in turn 
analyzes the signal, thereby determining the embedded commands. 

At the same time, the microphone 110 detects sounds in the 
vicinity of the television set 130 (step 152). The assumption is that the 



-13- 



10 



15 



20 



WO 00/70869 PCT/ILOO/00278 

major part of these sounds is generated by the television set, according to 
the viewed channels received from the broadcast center 124. 

The microphone 110 converts the detected sounds to an audio 
signal and provides it to the processor 104. The processor 104 processes 
the audio signal, thereby producing a respective digital signature 
therefrom (step 154). It is noted that a signature can be generated using 

analog techniques. 

According to the present example, the processor generates a 
record, which includes a plurality of data objects of various types (step 
156). A first type of data object includes information relating to the remote 
control command which was detected last. A second type of data object is 
the digital signature, which was produced from the moment of detection of 
that remote control command currently detected. A third type of data 
object includes a time stamp, which can include time and date information, 
respective of the moment in which the respective remote control 
command was detected, as well as the time period which relates to 
processed audio signal. It is noted that such a record can either include 
some of the above data objects or all of them. 

The processor 104 can store this record in storage unit 106. The 
processor 104 transmits the generated records to the monitoring center 
120 (step 158). It is noted that this transmission can either be in real time 
or at predetermined time intervals, after accumulation of a number of 
records in the storage unit 106. 

The monitoring center 120 includes a communication interface 
142, for communicating with the device 100, a communication interface 
146, for communicating with the broadcast center 124. a processor 144. a 
comparator 140 and a storage unit 148. The communication interface 142 
is connected to the comparator 140 and to the processor 144. The 
processor 144 is further connected to the storage unit 148. the comparator 
30 140. and to the communication Interface 146. 



25 
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The monitoring center 120 is connected to the device 100 via a 
communication link 122, and to the broadcast center 124, via a 
communication link 123. The broadcast center 124 provides the plurality 
of channels to the monitoring center 120. via the communication link 123. 
It is noted that communication links 122 and 123 can be either wireless, 
as in the present example, or wired. The processor 144 of the monitoring 
center 120 processes the received channels and produces a respective 
reference signature for each of them. According to one aspect of the 
present invention, the comparator 140 compares the digital signature 
received from device 100 v^mh each of the reference signatures (step 160), 
and detects compatibility with a specified one thereof, thereby determining 
a match with this reference signature (step 162). When a match is 
detected, then the comparator determines the channel associated with the 
reference signature as the channel, which is viewed by the user on the 

television set 130. 

According to another aspect of the present invention, the 
communication interface 142 provides selected objects of each one of the 
records received from the device 100. The processor 144 uses these 
objects to select a specific channel, and provide a respective reference 
signature to the comparator 140. The selection criteria can include 
references to the channel selected by the user according to the detected 
remote control user command, the time at which this command was 
detected, the time at which the record was created, the time at which the 
attached digital signature was generated by the device 100 and the like. 
The comparator 140 then compares the received digital signature with the 
selected reference signatures. 

When a match is detected, then comparator 140 provides a 
respective indication to the processor 144, which in turn stores this 
information in the storage unit 148. It is noted that the information stored 
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In Storage unit 148 can be processed according to a plurality of analysis 
methods to determine viewing habits and the like. 

Reference is now made to Figure 3, which Is a schematic 
illustration of a television set. referenced 230. a broadcast center. 

. referenced 252, a monitoring center referenced 250 and a television 
monitoring device generally referenced 200, constructed and operative In 
accordance with another preferred embodiment of the present Invention. 
The television set 230 Includes a video signal outlet 236. an audio outlet 
234 and a remote control signal detector 232. It Is noted that such audio 

) outlet can be an RCA outlet, an earphone outlet and the like. A remote 
control unit 240 Is associated with television set 230. and produces 
wireless signals, generally referenced 242, which represent specific 
orders received from a user. 

Monitoring device 200 includes a detector 208, which Is adapted 

5 to detect wireless transmissions 242, a processor 204. an audio interface 
210. a video interface 212. a storage unit 206 and a communication 
Interface 202. The processor 204 Is connected to detector 208, audio 
interface 210, video Interface 212, storage unit 206 and to communication 
interface 202. Device 200 demonstrates an aspect of the present 

!0 invention, in which the signal representing the channel viewed on the 
television set 230 is detected directly from the television set via standard 
outlets thereof. 

Audio interface 210 is further connected to audio outlet 234. 
Video interface 212 is further connected to video outlet 236. 

25 Communication interface 202 Is further connected to the monitoring center 
250. The broadcast center 252 is connected to the monitoring center 250 
via a communication link 256, and to the television set 230, via a 
communication link 254. It is noted that communication links 254 and 256 
can be either wire communication links or wireless communication links. 

30 as described herein above. 
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Reference is further made to Figure 4, which is a schematic 
illustration of a method for operating device 200, together with monitoring 
center 250, operative in accordance with a further preferred embodiment 
of the present Invention. 

In step 270, the device 200 detects a remote control device 
transmission. The remote control transmission Is generally detected by 
detector 208, which converts the received transmission signal Into digital 
data and provides It to the processor 204. 

In step 272, the device 200 receives a signal from the television 
set. This signal is respective of the currently viewed channel. Video 
Interface 212 detects a video signal, which Is provided by television set 
230. This video signal is respective of the currently viewed channel. It Is 
noted that this video signal can include a video steam as well as an audio 
stream. The audio interface 210 detects an audio signal, which is provided 
from the audio outlet 234. The audio signal Includes an audio stream. 

According to the present Invention, the device 200 has to 
receive at least one stream of information, video or audio. Accordingly, the 
audio interface can be used when the television set does not include a 
video outlet, or when the television set video outlet does not provide an 
audio stream. In any case, when the television set 230 includes a video 
outlet, then the audio Interface 210 is optional. 

The video Interface 212 and the audio interface 210 provide the 
detected streams to the processor 204. The processor 204 processes 
each of these streams and produces a digital signature therefrom (step 
274). It is noted that the processor 204 can produce a video digital 
signature from the video stream, an audio digital signature from the audio 
stream, or a combined digital signature from both streams. 

According to one aspect of the invention, the storage unit 206 
Includes a plurality of reference signatures, which are respective of a 
plurality of reference streams. It is noted that the storage unit 206 does 
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not contain these reference streams, rather an indication of the 
association between each of them and a respective reference signature. 

The processor 204 compares the digital signature with at least a 
selected group of the reference signatures and attempts to detect a match 
there between. This group of reference signatures can be selected 
according to a variety of selection criteria. The selection criteria can be the 
channel which is associated with the cun-ently detected remote control 
command; the time at which the streams that generated the digital 
signature was received; the most likely to be matched reference 
signatures according to recent user habit statistics, and the like. 

When the processor 204 detects that one of the reference 
signatures is compatible with the digital signature, it determines the 
reference stream, which is associated with the matched reference 
signature (step 278). Then, the processor 204 generates a record, which 
includes information relating to the respective reference stream, and a 
time stamp. Finally the processor 204 transmits this record to the 
monitoring center 250 (step 280). 

According to another aspect of the invention, at this point the 
processor 204 may generate a record, which is similar to the record 
produced by processor 104 (Figure 1). This record may include data 
objects relating to the digital signature, the detected remote control 
commands, a time stamp and the like. The processor 204 can store this 
record in the storage unit 206. The processor 204 transmits the record, 
either online or at predetermined time intervals, to the monitoring center 
250. The monitoring center includes means for comparing the digital 
signature embedded in the record to a plurality of reference signatures 
(step 276) and means for determining the reference stream, which is 
associated with a reference signature, which was detected as compatible 
with the digital signature 
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Reference is now made to Figure 5. which is a schematic 
Illustration of a device, generally referenced 300, constructed and 
operative in accordance with another preferred embodiment of the present 
invention. Figure 5 also includes schematic illustrations of a monitoring 
center, a broadcast center and a remotely controlled television set. 

Device 300 includes a processor 302, a communication 
interface 304, a tuner 306, a comparator 308, a television Interface 310, a 
remote control transmission detector 312 and a storage unit 314. 
Processor 302 is connected to communication interface 304, tuner 306, 
comparator 308, remote control transmission detector 312 and to storage 
unit 314. Comparator 308 is further connected to the television interface 
310, and to the tuner 306. 

A television set 330 is connected to a broadcast center 352. The 
broadcast center 352 is further connected to a monitoring center 350. The 
television set 330 includes a remote control unit 340, and a remote control 
detector 332 which is adapted to receive the remote control transmissions, 
generally referenced 344. 

The detector 312 is adapted to detect the remote control transmissions 
344, and to transfer digital representations of the user commands 
embedded therein, to the processor 302. The television interface 310 can 
include one or more units such as a video interface, for directly connecting 
to the television set 330. an audio interface for directly connecting to the 
television set 330, a microphone, for detecting sounds in the vicinity of the 
television set 330, a video camera pointed at the screen of the television 
set 330, and the like. Hence, the television interface 310 can be either 
physically connected to the television set 330, or wirelessly detecting 
signals which are produced therefrom. 

It is noted that the methods, which are presented in Figures 2 
and 4, are applicable for device 300. According to a further aspect of the 
invention, the device 300 can be used as a fast passive channel detection 
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device. Accordingly, the device 300 detects remote control transmissions, 
and extracts any channel change commands embedded therein, thereby 
determining a hypothesis of the channel selected by the user, operating 
the remote control unit. The processor 302 determines the channel 
according to a preset association table, which is stored in the storage unit 
314. The association table is determined according to the channel settings 
on the television set 330. Reference is now made to Table 1, which 
provides a combined presentation of the television set channel settings 
and the monitoring device association table. 



Table 1 



Entry (Station) Number 


Television Set Entry 


Monitoring Device Entry 


1 


Channel 21 


Channel 21 


2 


Channel 53 


Channel 53 








n 


Channel k 


Channel k 



The Entry Number column Indicates the same entry in the 
television set 330 station table, and in the monitoring device 300 
association table. Access to a specific entry is determined by the user, 
which operates the remote control device 340, to generate a respective 
command. When the detector 332 of television set 330 detects this 
command, then the television set 330 accesses the appropriate station 
entry, retrieves the channel information stored therein and tunes to that 
channel. At the same time, the detector 312 of the monitoring device 300 
detects the same remote control command, and provides this information 
to the processor 302, which in turn can send it to the monitoring center 
350. 

It is noted that the user is free to change the channel settings on 
the television set 330. Hence, the channel settings of the television set 
330 may not always be compatible with the association table of the 
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monitoring device 300. Reference is now made to Table 2. which provides 
a non compatible combined presentation of the television set channel 
settings and the monitoring device association table. 



Table 2 



Entry (Station) Number 


Television Set Entry 


Monitoring Device Entry 


1 


Channel 21 


Channel 21 


2 


Channel 68 


Channel 53 








N 


Channel k 


Channel k 



In the present example, the user programmed the second entry 
in the television set to include channel 68, while the settings of the second 
entry in the association table include channel 53. According to this aspect 
of the present invention, the device 300 includes a confirmation 
mechanism, which detects and corrects such mismatches. 

Reference is now made to Figure 6, which is a schematic 
illustration of a method for operating the device 300 of Figure 5, operative 
In accordance with a further preferred embodiment of the present 
invention. The detector 312 detects the channel change command and 
provides this information to the processor 302. The processor 302 
determines the entry associated with the detected command, accesses 
the association table and retrieves the channel associated with that entry. 
Then, the processor 302 provides a channel change command to the 
tuner 306 (step 360). 

The tuner 306, being connected to the same broadcast source 
as the television set 330, tunes to the selected channel, receives a 
broadcast signal, and provides a representation of that broadcast signal to 
the comparator 308 (step 362). It is noted that the representation can 
include either a video signal or an audio signal, which are extracted from 
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the broadcast signal, a digital signature (step 366) produced therefrom 
and the like. 

At the same time, the TV interface 310 detects a TV signal, 
which is produced by the television set 330 (step 364). As stated above, 
this signal can be detected in a plurality of methods such as detecting a 
video signal or an audio signal, which are received directly from the 
television set 330, detecting sounds in the vicinity of the television set 330, 
detecting a signal from a video camera pointed at the screen of the 
television set 330, and the like. 

The television interface 310 provides a representation of the 
detected TV signal to the comparator 308. It is noted that the 
representation can include either a video signal or an audio signal, which 
is extracted from the TV signal (depending on the source), a digital 
signature (step 368) produced therefrom, and the like. 

The comparator 308 compares components, which are derived 
from respective streams of the TV signal and the broadcast signal (step 
370). For example, the comparator can compare a digital signature 
produced from the video stream of the broadcast signal, with a respective 
digital signature, which is produced from the video stream of the TV 
signal. Similarly, the comparator can compare a digital signature produced 
from the audio stream of the broadcast signal, with a respective digital 
signature, which is produced from the audio stream of the TV signal. 

If the comparator 308 detects that the compared components 
are compatible, it provides this information to the processor 302. In turn, 
the processor 302 determines that the content of the respective entry in 
the association table is valid with respect to the respective entry in the 
channel settings of the television set 330. At this point, the processor 302 
transmits the confirmed channel selection of the user, to the monitoring 
center 350 via the communication interface 304 (step 374). 
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If the comparator 308 detects that the compared components 
are not compatible with each other, then it provides this information to the 
processor 302. It is noted that such a non compatibility indicates that the 
selected entry in the association table is not valid (see for example, entry 
No. 2 In Table 2). 

The processor 302 initiates a channel scanning procedure so as 
to determine the currently viewed channel. Accordingly, the processor 302 
specifies another channel and provides a respective channel change 
command to the tuner 306 (repeating from step 362). When a channel is 
detected to be compatible with the currently viewed channel, the 
processor 302 updates the selected entry in the association table so as to 
include the detected channel. According to the example of Table 2, the 
second entry in the association table is set to include channel 68. 

Some television sets are capable of presenting more than one 
channel at the same time. This function is called multiple channel 
presentation. Reference is now made to Figures 7A and 7B, which are 
schematic Illustrations of a television, generally referenced 390, which is 
adapted to provide multiple channel presentation. With reference to Figure 
7A, television set 390 presents the broadcast of a primary channel 
generally on the entire area of screen 394. In addition, a frame, generally 
referenced 392 is defined at a selected location on the screen 394, where 
the broadcast of a secondary channel is presented. This function is often 
called Picture in Picture (PIP). It is noted that such a setting enables the 
user to view other channels at a glance, without switching off the primary 
channel. On some television sets, the user may define the location and 
size of frame 392, anywhere on the screen 394. 

With reference to Figure 7B, television set 390 presents a 
multiple display of channels. Accordingly, a plurality of frames, generally 
referenced 396a. 396b. 396c, 396n-i and 396n, are defined on screen 394. 
The television set 390 presents a plurality of channels, generally denoted 
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channel 1, channel 2 to channel 9. It Is noted that the number of channels 
depends on the number of independent tuner modules within television 
set 390. Furthermore, any arrangement of frames can be displayed on 
screen 394 with respect to the specification of a selected television set. 

According to a further aspect of the present invention, there is 
thus provided a device which can monitor a plurality of viewed channels 
on a television set which is capable of PIP presentation. Reference is now 
made to Figure 8, which is a schematic illustration of a device, generally 
referenced 400, for monitoring multiple channels, constructed and 
operative in accordance with another preferred embodiment of the present 
invention. Figure 8 further includes a schematic illustration of a television 
set 430, a remote control unit 440, a monitoring center 450 and a 
broadcast center 452. 

Device 400 includes a communication interface 402, a 
processor 404, a multi-channel tuner 408, a television interface 410 and a 
remote control transmission detector 406. The processor 404 is connected 
to communication interface 402, multi-channel tuner 408, television 
interface 410 and to remote control transmission detector 406. 

The television interface 410 is connected to a video outlet 436 of 
television set 430. It is noted that television interface 410 can be replaced 
with a video camera which detects the entire image produced by the 
screen of television set 430. Television set 430 is connected to the 
broadcast center 452 through a broadcast inlet 438, via a communication 
link 454, which is also connected to multi-channel tuner 408. 

Processor 404 is a multi-tasking processor which is operative to 
run a plurality of processes at the same time. The multi-channel tuner 408 
includes a plurality of tuning modules (not shown) each being capable of 
tuning to a different channel. Hence the multi-channel tuner 408 provides 
a plurality of output channels at the same time. Device 400 is adapted to 
detect a multi-channel situation, determine the number and identity of 
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each of the displayed channels, and transmit this information to a remote 
monitoring center (reference 450). 

Reference is further made to Figure 9, which is a schematic 
Illustration of a method for operating monitoring device 400 of Figure 8, 
operative In accordance with a further preferred embodiment of the 
present invention. 

The detector 406 detects a remote control transmission 442, 
generated at the remote control unit 440, which includes a multi-channel 
display command (step 470). The detector 406 provides the detected 
command to the processor 404. The processor 404 analyzes the detected 
remote control command, thereby determining the number of channels 
which the user requested to be displayed, and a hypothesis for the 
identification of each of the channels (step 472). The processor 404 
produces a channel tuning command according to each of the 
hypotheses, and provides these channel tuning commands to the 
multi-channel tuner 408. The multi-channel tuner 408 assigns an 
Independent tuning module to each of the channel tuning commands, 
thereby providing a plurality of channel signals to the processor 404. The 
processor 404 processes each of the tuned channel signals (multiple 
channel analysis process module 416), thereby producing a digital 
signature, one for each of the channel signals (step 480). 

At the same time, the monitoring device 400 receives a TV 
signal from the television set 430 via the TV Interface 410 (step 474) and 
provides the TV signal to the processor 404. The processor 404 analyzes 
the TV signal (multiple frame detection process module 412) thereby 
determining the location of the frame, which presents each of the 
displayed channels (step 478) and produces a signature from each of the 
frames (step 482). It is noted that in a PIP case, the entire screen defines 
the frame of the primary channel (see Figure 7A). The processor 404 
initiates a comparison process (comparison process (1) module 414i and 
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comparison process (2) module 41 4n) for each pair, wliich includes a TV 
displayed channel and a tuned channel. Each of these modules 41 4i and 
41 4n performs a verification of the displayed channel with the respective 
tuned in channel (step 484). so as to confimi the hypothesis produced in 
step 472. When the hypothesis is confirmed, the processor 404 transmits 
the channel identification information to the remote monitoring center 450, 
via the communication interface 402 (step 488). Otherwise, the processor 
specifies another channel (step 486) and repeats from step 476. 

In accordance with a further aspect of the invention, a video 
capturing device is firmly attached to the monitored television set 430. 
According to this aspect, the capturing of the video image produced by the 
television set 430 is not affected by any position or orientation adjustment 
thereof. 

Reference is now made to Figure 10, which is an illustration of a 
television set and two capturing devices, generally referenced 500 and 
510, constructed and operative in accordance with a further preferred 
embodiment of the invention. 

Capturing devices 500 and 510 are video cameras, which are 
mounted on specially designed devices . Camera 500 is mounted on an 
arm 502, which is attached to the television set 520 via a base 504. It is 
noted that the base 504 can be attached to the body of the television set 
520, in a plurality of ways such as by means of adhesive materials, 

screws, and the like. 

The arm 502 places the camera 500 on top of the television set 
520, facing down. Hence, the camera 500 is not located within the line of 
sight of a potential viewer. 

Reference is further made to Figure 11 A, which is a schematic 
illustration of the view detected by the camera 500 (Figure 10). As can be 
seen, the detected picture is deformed in the iom of an upside trapezoid. 
Upper comers 522 and 524 of the screen (Figure 10) fomn the wider base 
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of the trapezoid (corners 522A and 524A), while lower corners 526 and 
528 of the screen (Figure 10) form the narrower base of the trapezoid 
(corners 526A and 528A). 

In accordance with the present invention, the detected image is 
reconstructed so as to obtain the original proportions of the screen of the 
television set 520, by allocating each pixel in the detected picture to a 
normalized matrix. After this step of reconstruction, any of the image 
processing procedures can be executed on the reconstructed image, such 
as signature generation, multi-picture detection and the like. 

With reference to Figure 10, camera 510 is mounted on the side 
of television set 520, on an arm 512 and a base 514. Reference is further 
made to Figure 1 1 B, which is a schematic illustration of the view detected 
by the camera 510 (Figure 10). As can be seen, the detected picture Is 
deformed in the form of a side-rotated trapezoid. The left corners 522 and 
526 of the screen (Figure 10) form the wider base of the trapezoid 
(corners 522B and 526B). while the right-sided comers 524 and 528 of the 
screen (Figure 10), fomi the narrower base of the trapezoid (corners 524B 
and 528B). 

It is noted that in this case, the connection procedure is very 
similar to the one shown in conjunction with Figure 11 A, where the image 
is nomrialized side-ways. 

It will be appreciated by persons skilled in the art that the 
present invention is not limited to what has been particulariy shown and 
described herein above. Rather the scope of the present invention Is 
defined only by the claims which follow. 
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CLAIMS 

1. Monitoring device comprising: 
a processor; 

a television remote control unit signal detector, connected to said 
5 processor; 

a signal capturing unit, connected to said processor; and 
a communication interface, connected to said processor, 
wherein said television remote control unit signal detector provides a 
detected television remote control unit signal to said processor, 
10 said signal capturing unit provides a detected television signal to 

said processor, and 
wherein said processor transmits data related to said detected 
television signal and said detected television remote control unit 
signal to a remote monitoring center, via said communication 
15 Interface. 

2. The device according to claim 1, wherein said television remote 
control unit signal detector is selected from the list consisting of: 

an infra-red detector; 
20 a radio frequency detector; and 

an ultrasound detector. 



3. The device according to claim 1, wherein said signal capturing unit is 
selected from the list consisting of: 
25 a microphone; 

a video camera; 

an audio interface; 

a video interface; and 

a broadcast signal interface. 
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The device according to claim 1 . wherein said data includes objects 
selected from the list consisting of: 

a representation of said detected television signal; 
a representation of said detected television remote control unit 

signal; 
a time stamp; 

a digital signature determined from said detected television 
signal; and 

channel identity information detemriined from said detected 
television remote control unit signal. 

The device according to claim 1, further comprising a storage unit 
connected to said processor, wherein said processor stores data In 
said storage unit for transmitting via said communication interface at 
a later time. 

The device according to claim 1 , wherein said signal capturing unit 
comprises a tuner, 

wherein said processor determines a broadcast channel from 
said detected television remote control unit signal and provides a 
command to said tuner to select said broadcast channel, 

wherein said tuner captures a broadcast signal transmitted on 
said broadcast channel, and 

wherein said processor determines a digital signature from said 
broadcast signal and transmits said digital signature via said 
communication interface. 

The device according to claim 1, further comprising a tuner, 
connected to said processor, and a comparator connected to said 
processor, said television interface and to said tuner, 



-29- 



wo 00/70869 PCT/ILOO/00278 

wherein said processor determines a broadcast channel from said 
detected television remote control unit signal, and provides a 
command to said tuner to select said broadcast channel. 

wherein said tuner captures a broadcast signal transmitted on said 
broadcast channel, and provides said broadcast signal to said 
comparator, 

wherein said signal capturing unit provides said detected television 
signal to said comparator, 

wherein said comparator compares respective properties of said 
detected television signal and said broadcast signal and 
provides a match result to said processor for verifying that said 
determined broadcast channel corresponds to said detected 
television signal. 

8. The device according to claim 1, wherein said signal capturing unit 
detects television video signal and provides said detected television 
video signal to said processor, 

wherein said processor detects a plurality of frames in said 
detected television video signal. 

9. The device according to claim 8, further comprising a multi channel 
tuner, connected to said processor, 

wherein said processor detemiines a plurality of broadcast 
channels from said detected television remote control unit signal 
and provides a command to said tuner to select said broadcast 
channels, 

wherein said tuner captures a plurality of broadcast signals 
transmitted on said selected broadcast channels, and provides 
said broadcast signals to said processor, and 
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wherein said processor compares between an image provided in 
each of said frames and a selected one of said broadcast 
signals. 

10. Method for detecting user selection of television channel comprising 
the steps of: 

detecting a remote control device transmission; 
receiving a television signal from a television set; and 
processing said television signal thereby generating a signature 
therefrom. 

11. The method according to claim 10, further comprising the step of 
producing a record including a plurality of data objects, wherein at 
least one of said data objects comprises said signature, and at least 
another of said data objects comprises a time stamp. 

12. The method according to claim 11. further comprising the step of 
transmitting said record to a remote control center. 

13. The method according to claim 10, further comprising the step of 
comparing said signature with at least one of a plurality of reference 
signatures, thereby determining a compatible reference signature 
therefrom. 

14. The method according to claim 13, further comprising the step of 
determining a stream corresponding with said compatible reference 
signature. 
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15. The method according to claim 14, further comprising the step of 
transmitting information relating to said determined stream, to a 
remote monitoring center. 

5 16. The method according to claim 14, further comprising the step of 
storing information relating to said detennined stream. 

17. The method according to claim 10, further comprising the step of 
storing said signature. 

10 

18. The method according to claim 10, further comprising the steps of: 

detecting a video signal from a specific channel; 
processing said video signal thereby generating a channel 
signature therefrom; and 
15 comparing said channel signature with said signature, thereby 

verifying that said specific channel corresponds to said television 
signal. 

19. The method according to claim 18, further comprising the step of 
20 selecting said specific channel according to said remote control 

device transmission. 

20. The method according to claim 19, further comprising the step of 
selecting a different channel when said specific channel is not 

25 compatible with said television signal. 

21. The method according to claim 19, further comprising the step of 
transmitting parameters relating to said specific channel, when said 
specific channel is compatible with said television signal. 

30 
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22. The method according to claim 1 0, further comprising the steps of: 

analyzing said television signal thereby determining a plurality of 
frames; and 

retrieving a frame television signal from each said frames. 

23. Method for detecting user selection of television channel comprising 
the steps of: 

receiving a television signal from a television set; 
detecting a plurality of channel images within said television 
signal; and 

processing said television signal thereby generating a signature 
for each said channel images. 

24. The method according to claim 23, further comprising the steps of: 

detecting a remote control device transmission; and 

analyzing said remote control device transmission, so as to 

extract multiple channel commands embedded therein, 

wherein said multiple channel commands are further used in 

said step of detecting said plurality of channel images, 

25. The method according to claim 23, further comprising the step of 
producing a plurality of records, each said records Including a 
plurality of data objects, wherein at least one of said data objects 
comprises said signature, and at least another of said data objects 
comprises a time stamp. 

26. The method according to claim 25, further comprising the step of 
transmitting said records to a remote control center. 
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27. The method according to claim 23, further comprising the step of 
comparing each said signatures with at least one of a plurality of 
reference signatures, thereby determining at least one compatible 
reference signature, therefrom. 

28. The method according to claim 27, further comprising the step of 
determining a stream corresponding with said at least one compatible 
reference signature. 

29. The method according to claim 28, further comprising the step of 
transmitting information relating to said determined stream, to a 
remote monitoring center. 

30. The method according to claim 29, further comprising the step of 
storing information relating to said determined stream. 

31. The method according to claim 23, further comprising the step of 
storing said signature. 

32. The method according to claim 23, further comprising the steps of: 

detecting video signals from a plurality of specific channels; 

processing said video signals, thereby generating a channel 
signatures therefrom; and 

comparing said channel signatures with said signature, thereby 
verifying that at least one of said specific channels corresponds to at 
least a portion of said television signal. 

33. The method according to claim 32, further comprising the step of 
selecting said specific channels according to said remote control 

I device transmission. 
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34. The method according to claim 33. further comprising the step of 
selecting a different channel when said at least one of said specific 
channel is not compatible with at least a portion of said television 

5 signal. 

35. The method according to claim 33, further comprising the step of 
transmitting parameters relating to said specific channel, when said 
specific channel is compatible with at least a portion of said television 

10 signal. 

36. Monitoring device according to any of the claims 1-9 substantially as 
described herein above. 

15 37. Monitoring device according to any of the claims 1-9 substantially as 
illustrated in any of the drawings. 

38. A method according to any of the claims 10-22 substantially as 
described herein above 

20 

39. A method according to any of the claims 10-22 substantially as 
illustrated in any of the drawings. 

40. A method according to any of the claims 23-35 substantially as 
25 described herein above. 

41. A method according to any of the claims 23-35 substantially as 
illustrated in any of the drawings. 

30 
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DETECTING A REMOTE CONTROL DEVICE TRANSMISSION 



150 



RECEIVING A SIGNAL FROM A TELEVISION SET 



152 



PROCESSING THE SIGNAL. THEREBY GENERATING A 
SIGNATURE THEREFROM 



54 



STORING A RECORD CONTAINING THE SIGNATURE WITH 
INFORMATION RELATED TO THE REMOTE CONTROL DEVICE 

TRANSMISSION 



156 



T 



TRANSMITTING THE RECORD TO A REMOTE MONITORING 

CENTER 



158 



7 



COMPARING THE SIGNATURE TO A PLURALITY OF REFERENCE 

SIGNATURES 



160 



DETERMINING THE STREAM WHICH WAS PROVIDED BY THE 

TELEVISION SET 

1 

162 
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DETECTING A REMOTE CONTROL DEVICE TRANSMISSION 
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SIGNATURE THEREFROM 



COMPARING THE SIGNATURE TO A PLURALITY OF REFERENCE 

SIGNATURES 
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DETERMINING THE STREAM WHICH WAS PROVIDED BY THE 

TELEVISION SET 
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DETERMINED STREAM TO A REMOTE MONITORING CENTER 

\ 
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DETECTING A REMOTE CONTROL DEVICE TRANSMISSION 
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RECEIVING A TV SIGNAL 
FROM A TELEVISION SET 
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PROCESSING THE TV 
SIGNAL, THEREBY 
GENERATING A TV 
SIGNATURE THEREFROM 
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DETECTING A VIDEO SIGNAL 
FROM A SPECIFIC CHANNEL 
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THAT THE DETECTED VIDEO SIGNAL IS 
COMPATIBLE WITH THE ONE CURRENTLY 
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